Skip to content

Add the DC-EGM solver (discrete-continuous endogenous grid method)#390

Draft
hmgaudecker wants to merge 203 commits into
mainfrom
feat/dcegm
Draft

Add the DC-EGM solver (discrete-continuous endogenous grid method)#390
hmgaudecker wants to merge 203 commits into
mainfrom
feat/dcegm

Conversation

@hmgaudecker

@hmgaudecker hmgaudecker commented Jun 19, 2026

Copy link
Copy Markdown
Member

Draft. Stacked on #391 (feat/type-local-continuation-v) → main. (The earlier
#388/#389 stack has been consolidated into this branch.)

Adds the DC-EGM solver (discrete-continuous endogenous grid method) behind the
Solver ABC seam. It inverts the Euler equation on an exogenous end-of-period grid
instead of grid-searching the continuous action, with a FUES upper envelope to drop
dominated candidates. Selected per regime via solver=DCEGM(...); GridSearch (the
default) is untouched, and the engine dispatches polymorphically — no fork on solver
type.

Commit structure (reviewable in steps)

  1. DC-EGM solver implementation (squashed) — the kernel: Euler inversion, the FUES
    envelope, the closed-form constrained segment, multi-target continuation, and
    asset-row mode; wired through the ABC DCEGM.validate / build_period_kernels.
  2. Stochastic-node expectation in lax.scan blocks — a memory knob for the child
    stochastic-node mesh.
  3. FUES provenance + credits (3 commits) — NOTICE/license for the FUES derivation and
    an acknowledgments page.
  4. Modular split of step.py (5 pure-move commits) — regime_introspection
    kernel_scopecontinuationstep_coreasset_row, taking the monolith from
    ~2900 to ~800 lines so the simple single-post-state path reads cleanly apart from the
    asset-row and continuation machinery.
  5. bind_continuation — collapse the continuation behind one bound
    savings → (expected_value, expected_marginal) callable (ContinuationPlan).
  6. Beartype + test fixes (2 commits) — the contract dataclasses are beartyped by the
    package claw, so cyclic field types use the two-definition alias pattern (precise for
    ty, bare container at runtime); plus a valid-DCEGM seam build test.
  7. Seam Layer 2 — a uniform PeriodKernel/KernelResult adapter; the solve loop
    calls one kernel and branches only on optional generic outputs, not on solver type.
  8. Streaming refine-to-query — fold the asset-row single-query interpolation into the
    FUES scan (refine_to_bracket + publish_node_from_bracket), removing the n_pad
    intra-node envelope scratch. Single-post-state still publishes the full refined
    envelope as its carry, unchanged.

Solvers and accuracy work built on the DC-EGM seam

On top of the kernel, this branch adds the prime-time solvers and the Dobrescu–Shanker
benchmark reproductions:

  • NEGM (nested EGM) and the 1-D RFC / MSS / LTM upper-envelope backends
    (DCEGM(upper_envelope=...)), plus the 2-D RFC/G2EGM two-asset foundation for the
    DS-2024 pension comparison.
  • DS-2026 Applications 1/2/3 reproduced (retirement, continuous-housing NEGM +
    EGM-FUES, discrete-housing-with-tax); full paper-scale matrix run on gpu-01.
  • Jit internal functions on a per-period level #129 — branch-aware VFI oracle (tests/solution/_branch_aware_vfi_oracle.py): pins
    the App.2 (S,s) EGM-vs-VFI gap to a comparator-ordering term that is identically zero
    on the scored interior, correcting the earlier attribution; the tolerated interior gap
    is DC-EGM durable-corner discretization. Standard-mode oracle == brute to 8e-5.
  • DS-2024 housing RFC-vs-NEGM comparison (ds2024_housing*.py,
    benchmarks/ds_replication/ds2024_housing_comparison.py): NEGM (nested continuous
    housing) vs RFC/FUES (discrete-housing DC-EGM, the source's per-housing-column 1-D
    rooftop cut). Reproduces the paper headline — RFC runtime < NEGM runtime — against a
    grid-search VFI oracle (faithful at delta = 0; the paper's delta = 0.10 keeper
    depreciation awaits a shared housing-axis carry extension).

Validation

  • Full gpu-01 oracle battery (Tesla V100, -n 4, on a free GPU): 1405 passed, 15
    skipped, 2 xfailed
    . Covers the DC-EGM-vs-brute value-function-parity battery, the
    19-case refine-to-query unit equivalence (streamed ≡ row-then-interp), and the Layer-2
    distinct-compiled-cores regression.
  • ty and prek clean.

Pending (does not block reviewing the engine)

  • DS-2024 housing delta = 0.10 fidelity: the keeper depreciates the held stock off
    the housing grid; the strict-identity NEGM keeper needs a housing-axis carry extension.
  • ACA-scale memory (Gate C): refine-to-query's n_pad runtime win is only visible at
    ACA scale — the in-suite oracles are too small to exercise it; handed to the aca side.
  • Throughput vs the brute-force solver at matched accuracy (Gate E) is a downstream
    experiment this work unlocks, not settled here.

🤖 Generated with Claude Code

hmgaudecker and others added 3 commits June 17, 2026 15:01
Route the existing brute-force grid search through a per-regime solver
configuration and a builder registry, with no change to the numerics.

- `Regime.solver: BruteForce | DCEGM` (default `BruteForce()`), exported from
  `lcm` alongside the `DCEGM` configuration class.
- `_lcm.solution.registry`: `SolverBuildContext`, `SolverKernels`, the
  `SolverKernelBuilder` protocol, and `_build_brute_force_kernels` (the former
  `_build_max_Q_over_a_per_period`), dispatched on `type(regime.solver)` via
  `SOLVER_KERNEL_BUILDERS`.
- `DCEGM` is published as the final configuration surface (fields + field
  validation), but its engine is not yet wired in; a regime requesting it is
  rejected at model build with `NotImplementedError`.

Behavior-preserving: the full test suite passes unchanged and an explicit
`BruteForce()` yields the same value function as the default.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Points the benchmark feature's aca-model at feat/dcegm-solver's tip — main's
audit fixes + the DC-EGM solver + smooth-share eligibility — the version the
solver-seam work is developed against.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the `type(solver)`-keyed builder registry with a polymorphic `Solver`
ABC. The engine now calls `solver.validate(context)` then
`solver.build_period_kernels(context)` — no `SOLVER_KERNEL_BUILDERS` dict, no
`BruteForce | DCEGM` union, no standalone DC-EGM guard.

- `_lcm/solution/contract.py` (new): the `Solver` ABC (abstract
  `build_period_kernels`, default no-op `validate`), `SolverBuildContext`, and
  `SolverKernels`. An engine leaf — imports nothing that reaches `lcm.solvers`,
  so the façade can re-export it without an import cycle.
- `_lcm/solution/solvers.py` (new): `GridSearch(Solver)` (the relocated
  grid-search builder, with function-local `jax`/`get_max_Q_over_a` imports) and
  `DCEGM(Solver)` (the published config; `validate` raises the not-yet-available
  guard, so a regime requesting it is rejected at model build).
- `lcm/solvers.py` → thin re-export façade; `registry.py` deleted; the
  `processing` dispatch and `Regime.solver` field updated.
- Rename the default solver `BruteForce` → `GridSearch` (more descriptive;
  alpha permits the break).

Faithful to dcegm-solver-seam-abc-design.md (ABC, not Protocol). Layer 2
(generic KernelResult) does not apply here — the stub seam has no EGM fork.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@review-notebook-app

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@read-the-docs-community

read-the-docs-community Bot commented Jun 19, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown

Benchmark comparison (main → HEAD)

Comparing 28d7d93a (main) → 13fdf829 (HEAD)

Benchmark Statistic before after Ratio Alert
aca-baseline execution time 13.527 s 14.470 s 1.07
peak GPU mem 588 MB 588 MB 1.00
compilation time 374.68 s 373.04 s 1.00
peak CPU mem 7.03 GB 6.97 GB 0.99
aca-baseline-debug execution time 58.332 s 58.396 s 1.00
peak GPU mem 587 MB 587 MB 1.00
compilation time 440.03 s 439.58 s 1.00
peak CPU mem 7.76 GB 7.79 GB 1.00
Mahler-Yum execution time 4.697 s 4.603 s 0.98
peak GPU mem 520 MB 520 MB 1.00
compilation time 11.33 s 11.35 s 1.00
peak CPU mem 1.58 GB 1.58 GB 1.00
Precautionary Savings - Solve execution time 24.2 ms 24.0 ms 0.99
peak GPU mem 8 MB 8 MB 1.00
compilation time 1.57 s 1.59 s 1.01
peak CPU mem 1.15 GB 1.16 GB 1.00
Precautionary Savings - Simulate execution time 61.9 ms 62.0 ms 1.00
peak GPU mem 157 MB 157 MB 1.00
compilation time 3.58 s 3.61 s 1.01
peak CPU mem 1.33 GB 1.32 GB 0.99
Precautionary Savings - Solve & Simulate execution time 94.9 ms 90.1 ms 0.95
peak GPU mem 566 MB 566 MB 1.00
compilation time 4.75 s 4.80 s 1.01
peak CPU mem 1.31 GB 1.31 GB 1.00
Precautionary Savings - Solve & Simulate (irreg) execution time 202.7 ms 198.2 ms 0.98
peak GPU mem 2.18 GB 2.18 GB 1.00
compilation time 5.11 s 5.04 s 0.99
peak CPU mem 1.37 GB 1.37 GB 1.00
IskhakovEtAl2017DCEGMSimulate execution time 211.9 ms
compilation time 4.28 s
peak CPU mem 1.69 GB
IskhakovEtAl2017DCEGMSolve execution time 1.904 s
compilation time 7.08 s
peak CPU mem 1.59 GB
IskhakovEtAl2017Simulate execution time 200.6 ms 204.7 ms 1.02
compilation time 4.18 s 4.23 s 1.01
peak CPU mem 1.29 GB 1.29 GB 1.00
IskhakovEtAl2017Solve execution time 44.5 ms 48.1 ms 1.08
compilation time 0.70 s 0.70 s 1.00
peak CPU mem 1.15 GB 1.15 GB 1.00
IskhakovEtAl2017DCEGMSimulateGpuPeakMem peak GPU mem 282 MB
IskhakovEtAl2017DCEGMSolveGpuPeakMem peak GPU mem 0 MB
IskhakovEtAl2017SimulateGpuPeakMem peak GPU mem 281 MB 281 MB 1.00
IskhakovEtAl2017SolveGpuPeakMem peak GPU mem 67 MB 67 MB 1.00

hmgaudecker and others added 21 commits June 21, 2026 18:09
The full discrete-continuous endogenous grid method, re-rooted as a single
commit on top of the solver-selection seam (PR #388). The tree reproduces the
reviewed feat/dcegm tip exactly, minus two root-level audit scratch files.

On top of the seam this adds the `_lcm.egm` engine, DC-EGM build-time
validation, the `_build_dcegm_kernels` builder and the EGM-specific
SolverBuildContext / SolverKernels / SolutionPhase fields, and replaces the
seam's build-time `NotImplementedError` guard with the real solver. To be
split into reviewable sub-PRs at review time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The child stochastic-node expectation built the full node-stacked
`node_values`/`node_marginals` arrays and reduced them with a weighted sum
downstream of the map. So `stochastic_node_batch_size` (a `lax.map`) shed only
the per-node gather working-set while still materialising the full
`(..., n_nodes)` output — half a fix, and at large scale the residual stack can
bind and even raise peak.

Fold the weighted sum into a `lax.scan` carry instead: each block reads only its
nodes and accumulates the running expectation, so neither the per-node gather
nor the full node-stacked output is materialised. `0` (or a size >= the mesh)
keeps the single fused vmap + reduction.

The block reduction reorders the floating-point adds, so the value function
matches the fused solve to tight numerical tolerance rather than bit-identically;
the node-batch invariance test asserts `allclose` at rtol/atol 1e-9.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New `docs/credits.md` (linked into the docs nav and the README), crediting
  the methods pylcm builds on (Carroll EGM; Iskhakov–Jørgensen–Rust–Schjerning
  DC-EGM; Dobrescu–Shanker FUES; Fella generalized EGM; the Druedahl /
  Druedahl–Jørgensen / Dobrescu–Shanker multidimensional-EGM methods behind the
  reserved upper-envelope backend), the discretization methods (Tauchen,
  Rouwenhorst, Kopecky–Suen, Fella–Gallipoli–Pan), the replicated example models
  shipped in pylcm (IJRS 2017, Mahler–Yum 2024), QuantEcon, and the open-source
  ecosystem (OSE, NumEconCopenhagen, akshayshanker, JeppeDruedahl, fediskhakov).
- Add the missing BibTeX entries (Carroll 2006, Fella 2014, Dobrescu–Shanker
  FUES 2022 and RFC 2024, Druedahl 2021, Druedahl–Jørgensen 2017).
- Fix the Mahler & Yum (2024) page range in references.bib: 1697–1733
  (Econometrica 92(5)), not 1307–1343.

Every citation verified against the journal/DOI or the authors' repositories.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pylcm's Fast Upper-Envelope Scan was adapted from the
`OpenSourceEconomics/upper-envelope` package (Apache-2.0, © The Upper-Envelope
Authors) and since substantially modified. Both projects are Apache-2.0, so the
derivation is permitted; this records the upstream attribution and the
modification notice (Apache-2.0 §4(b)/(c)) in the `fues.py` header and the
Credits page, which previously credited only the method's paper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pylcm's Fast Upper-Envelope Scan is adapted from OpenSourceEconomics/upper-envelope
(Apache-2.0). Add a root `NOTICE` recording that derivation and the upstream
copyright, and vendor the upstream license text verbatim at
`licenses/upper-envelope-LICENSE`. The in-source attribution in `fues.py` (which
ships in the wheel via `src/`) already carries the modification notice; these
files add the conventional repo/sdist-level attribution.

No other part of the codebase derives from third-party source — the DC-EGM kernel
and the NEGM Phase-0 toys are written against pylcm's own API — so only the
upper-envelope derivation is recorded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the pure spec-introspection helpers (discrete/passive/process state
names, child state name, child discrete actions, child resources function and
its arg names, regime-function concatenation) out of step.py into
egm/regime_introspection.py. These read only a regime's static spec and hold
no kernel state, so they form the dependency leaf that the kernel build, the
continuation subsystem, and the kernel-scope checks all import from — breaking
the otherwise-circular dependency between step.py and the scope checks that
move next. Behavior-preserving pure move.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the four `_find_unsupported_*` functions — the build-time checks that
name features outside the kernel's current scope — out of step.py into
egm/kernel_scope.py. They read the processed kernel-build context (period
carry targets, qualified flat params, the regime's VInterpolationInfo), which
the model-time `validation` module does not have, so they belong with the
kernel build rather than with the model-construction validators. step.py keeps
calling `_find_unsupported_feature` to install the raising step. Behavior-
preserving pure move.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the multi-target continuation machinery — target splitting, the per-child
carry reader, per-row resources queries and gradients, the stochastic-node
expectation, passive-state blending, discrete-choice aggregation (hard max or
EV1 logsum), and the child-read build — out of step.py into egm/continuation.py.
The core EGM step and the asset-row step keep calling `_get_child_carry_reader`
/ `_build_child_reads` / `get_egm_continuation_targets`, so step.py shrinks by
~870 lines and the textbook Euler inversion no longer sits in the same file as
the multi-target, passive-state, taste-shock, and stochastic-node logic.
Behavior-preserving pure move; the bound continuation contract is a follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the certifiable core algorithm — _EgmKernelPieces, _get_solve_one_combo,
_compute_nodes_over_savings, _get_compute_node, _compute_constrained_candidates,
_publish_V_and_carry_rows, and the constrained-offset constant — out of step.py
into egm/step_core.py. It reads the expected continuation through
continuation's per-target carry reader and depends on nothing in asset-row mode
or the kernel orchestration. step.py keeps the orchestration (build, dispatch,
combo map) and the asset-row path (extracted next). The one test importing
_publish_V_and_carry_rows from step is repointed to step_core. Behavior-
preserving pure move.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the per-Euler-node solve — _get_solve_one_combo_asset_rows, its
differentiated continuation closure _get_expected_continuation_value, and the
scalar-query publisher _publish_node_V_and_policy — out of step.py into
egm/asset_row.py. It maps step_core's single-post-state pipeline over the Euler
grid and consumes continuation's per-target reader; step.py keeps only the
orchestration (build, dispatch, combo map, feasibility, raising step). This
completes the split: the textbook core (step_core) and the asset-row extension
(asset_row) are now separate, certifiable modules. Behavior-preserving pure
move; the streaming refine-to-query feature lands here next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collapse the multi-target continuation aggregation into `ContinuationPlan`
(build-time statics) plus `bind_continuation`, which binds the plan to one
combo pool and the next period's carries and returns a
`savings -> (expected value, expected marginal)` callable. The textbook step
(`_get_compute_node`) and the asset-row step (`_get_expected_continuation_value`)
read only that bound callable instead of each re-deriving the regime-transition
probabilities, child carry readers, and per-target aggregation.

Behavior-preserving: the bound callable performs the identical aggregation, and
it evaluates the probabilities and child reads from the combo pool *inside* the
builder, so the asset-row Euler-slot gradient still carries the direct
dP/da . EV terms (precomputing them outside the differentiated closure would
silently drop them).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the GridSearch rename.

The re-rooted `SolverBuildContext` / `SolverKernels` are beartyped by the package
claw, so under PEP 649 their field annotations resolve to real objects when a
context is constructed at model build. `UserRegime` and `VInterpolationInfo`
(whose module imports `lcm.regime`) close an import cycle through the
`lcm.solvers` façade, so each is referenced via a two-form alias — precise for ty
under `TYPE_CHECKING`, a bare container for the beartype claw at runtime. The
other engine types import normally.

The `BruteForce` → `GridSearch` rename had not reached the DC-EGM oracle test
modules; propagate it, including the validation-message `match=` string.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`test_model_with_dcegm_solver_builds` asserts that selecting the DC-EGM solver
builds the model rather than being rejected, but it placed the solver on the
stock iskhakov `working_life` regime, which lacks the resources, post-decision,
and inverse-marginal-utility functions the DC-EGM contract requires — so
validation correctly rejected it. Point the test at `build_dcegm_model`, whose
regimes satisfy the contract.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r 2)

Remove the residual solver-type fork in the backward-induction loop. The
loop no longer branches `if regime.solution.egm_step is not None`; every
regime — grid search or DC-EGM — exposes one uniform per-period adapter that
wraps its shared jitted core, calls it with the solver's own argument layout,
and assembles a `KernelResult(V_arr, carry, sim_policy)` outside JIT. The only
branches left in the loop are on the optional generic outputs
(`result.carry is not None` / `result.sim_policy is not None`), never on
solver type.

Contract (`_lcm/solution/contract.py`):
- Add `KernelResult` (frozen dataclass) and `PeriodKernel` (runtime_checkable
  structural Protocol exposing `core`, `build_lower_args`, `with_fixed_params`,
  `__call__`).
- Reshape `SolverKernels` → `SolutionKernels(period_kernels, continuation_template)`,
  dropping the `max_Q_over_a` / `egm_step` / `egm_carry_template` /
  `egm_reachable_targets` quartet.
- Name the continuation channel solver-agnostically via `ContinuationPayload`
  (= `EGMCarry` today).

Solvers (`_lcm/solution/solvers.py`):
- `GridSearch`/`DCEGM.build_period_kernels` build per-period adapter closures
  (`_GridSearchPeriodKernel` / `_DCEGMPeriodKernel`) over the existing jitted
  core; the EGM core's reachable-carry filter and param projection move onto
  the DC-EGM adapter.

Engine wiring:
- `SolutionPhase` carries `period_kernels` + `continuation_template` (plus a
  `solves_via_dcegm` property); the cyclic `PeriodKernel` annotation uses the
  TYPE_CHECKING-precise / runtime-widened two-definition pattern so the
  beartype claw resolves the dataclass field at construction.
- The terminal-continuation publisher is composed engine-side
  (`_TerminalCarryPeriodKernel` in regime_building/processing.py) as an
  output decorator, not inside `GridSearch`.
- `model_processing._partial_fixed_params_into_regimes` binds fixed params via
  each adapter's `with_fixed_params`; `simulation/additional_targets.py` reads
  `solves_via_dcegm`.

Compilation reuse and the loop's memory/host-offload/gc discipline are
preserved: only the shared core is jitted and identity-deduped (by
`id(Q_and_F)` / function identity), no adapter is jitted per period, and the
solve loop's device eviction / carry-buffer release / reachable-carry subset
logic is untouched (dispatch re-routed only). Add
`test_period_kernels_sharing_a_config_reuse_one_compiled_core` asserting the
distinct-compiled-core count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In asset-row mode the per-node solve refines a full NaN-padded upper
envelope of static length n_pad and interpolates it at exactly one
query (resources_at_node) to publish a scalar (V_node, policy_node),
then discards it. Batched over combos x Euler-nodes, that n_pad scratch
axis is the binder that OOMs large solves.

Fold the single-query interpolation into the upper-envelope scan so the
n_pad rows never materialize:

- Extract `_interp_between_nodes` in interp.py — the pure two-bracket-node
  arithmetic plus the Hermite correction, shared by `interp_on_prepared_grid`
  and the streamed path. The existing interp unit tests pin it (behavior-
  preserving). This guarantees the streamed value cannot diverge from
  row-then-interp: only which two nodes differs.
- Add `refine_to_bracket` + `QueryBracket` in fues.py — a new scan driver
  reusing `_inspect_candidate` verbatim, changing only the emission sink: each
  step's up-to-3 emitted points fold into an O(1) bracket-capture carry
  (first/second, rolling prev/last, lo = latest emitted with grid <= q, hi =
  first with grid > q, running counts), no [n_input,3] stack and no [n_pad]
  rows. Post-scan, the bracket reproduces
  clip(searchsorted(side="right"), 1, max(n_kept-1, 1)) node-for-node: the
  side="right" tie-break at a duplicated kink (right copy wins the lower slot),
  the below-first clamp (first, second), and the at-or-above-last clamp
  (second-last, last; single-live falls back to the NaN-padded slot like the
  row path). Geometry only — no utility, borrowing limit, or floor.
- Add `publish_node_from_bracket` in asset_row.py — the scalar-bracket
  counterpart of `_publish_node_V_and_policy`: the shared `_interp_between_nodes`
  with the value Hermite slope = grad(utility) at the two bracket policies only,
  plus the constrained floor and the n_kept > n_pad overflow poison, preserved
  identically. Wire the asset-row node solve to refine_to_bracket ->
  publish_node_from_bracket and drop the n_pad envelope from that path.

Scope: asset-row mode only. Single-post-state mode publishes the refined
envelope AS its inter-period carry (queried later at many parent points), so
step_core's single-post-state path keeps calling `refine` unchanged.

The new equivalence test pins the streamed pair against refine +
_publish_node_V_and_policy for the same candidates + query, to fp tolerance:
smooth, kinked, multi-crossing, all-dead, single-live, overflow, and query
below-first / above-last / exactly on a duplicated kink abscissa.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backward-induction loop forced a `gc.collect()` every period to return the
device pool from a rolled-off continuation carry (a registered pytree CPython
frees only on a cyclic collection). A grid-search period rolls no such carry, so
the collection bought nothing there while its fixed cost dominated the warm solve
of small / many-period models — deteriorating those benchmarks by up to ~25x
(the Iskhakov-et-al grid-search warm solve: 1.01s -> 0.35s).

Gate the collection on the loop's own per-period carry output
(`period_egm_carries`), empty exactly when no carry rolled. The backward-
induction loop stays backend-agnostic — it does not fork on solver type or read
`solution.solves_via_dcegm`; the clean separation of the solve backends is a
separate design task.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Dobrescu-Shanker housing keeper (no-house-trade) regime is a plain
1-D DC-EGM model: liquid assets is the Euler state, consumption the
continuous action, and housing a passive continuous state whose service
flow `alpha * log(housing)` is read directly by utility.

The cornerstone test locks in that the DC-EGM envelope-condition guard
admits utility reading a *passive* continuous state (it forbids only the
Euler state) by calling `validate_dcegm_regimes` directly on the
finalized keeper regimes — no kernel trace, no solve. The full GPU solve
is gated off the local box.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-0 of the NEGM / multidimensional DC-EGM build (general 2-asset capability):

- `negm_phase0/brute_feasibility_probe.py`: sweeps a 2-continuous-state
  (liquid + illiquid) + 2-continuous-action + income-shock lifecycle solved with
  BruteForce, reporting wall-clock and GPU peak. Finding: brute is single-solve
  feasible at fine grids (~4-9 s, <600 MB on a V100 up to 2.5 B combos/period) —
  the "brute breaks at Laibson scale" premise does not hold; NEGM is an
  accuracy/throughput/generality play, not a feasibility rescue.
- `negm_phase0/kinked_toy_oracle.py`: the smallest 2-asset toy carrying the
  Laibson frictions (credit-card rate kink at a^X=0, withdrawal penalty at Iz=0,
  Z>=0 floor, u(C+iota*Z)), brute-solved as the V-parity oracle for a future NEGM
  prototype.
- `negm_phase0/negm-phase0-findings.md`: brute-feasibility table, the NEGM
  nesting derivation with the P1 kink-candidate handling, build-vs-buy, and the
  gate recommendation (build for the general capability; reframe the motivation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…del.

Implements P2 of the Dobrescu–Shanker replication: the NEGM solver as a
third `Solver` on the ABC seam, composing an inner 1-D `DCEGM` solve of the
consumption-savings problem with an outer deterministic grid search over a
durable/illiquid margin.

- `NEGM(Solver)` in `_lcm/solution/solvers.py` (re-exported from
  `lcm.solvers` and `lcm`): frozen dataclass with `inner: DCEGM`,
  `outer_action`, `outer_post_decision`, `outer_grid`, optional
  `outer_no_adjustment_candidate`. `__post_init__` guards reject a stochastic
  outer grid and margins that coincide with the inner ones
  (`RegimeInitializationError`), mirroring DCEGM's `_fail_if_*` helpers.
- `_NEGMPeriodKernel`: non-jitted outer adapter over the shared jitted inner
  DC-EGM core; sweeps the outer grid plus the per-node no-adjustment candidate,
  binds the outer post-decision into the regime's flat params per node, and
  collapses the outer axis by `max`. Static only (ty + prek green); not run.
- `validate_negm_regimes` in `_lcm/egm/negm_validation.py`, wired into the same
  model-build path as `validate_dcegm_regimes`. Fail-loudly contract
  (`ModelInitializationError`, each message naming the alternative solver): no
  outer margin → use `DCEGM`; outer margin coincides with the inner margin →
  reject; coupled-2-Euler (outer post-decision feeds the inner Euler law /
  savings-stage pool, or a non-additively-separable utility cross-term) → use
  the 2-D EGM foundation; taste-shocked discrete choice → reject (ordering).
- Kinked-toy NEGM model (`tests/test_models/negm_kinked_toy.py`) and the G1
  parity test (`tests/solution/test_negm_kinked_toy.py`, skipped — GPU-only).
- Unit tests: 6 config-guard cases, 8 contract-validator cases (construct and
  assert-raise, no solve).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DS housing-keeper DC-EGM probe gains a brute-force (GridSearch) twin and a
DC-EGM-vs-brute value-parity test, mirroring tests/solution/test_egm_passive.

The brute twin solves the same keeper economics without the Euler/passive
split: housing is a regular fixed continuous state, consumption a grid-searched
action, the liquid-asset law of motion reads consumption directly, and a
borrowing constraint floors post-decision liquid assets at the borrowing limit
(the floor of the DC-EGM savings grid). build_working_regime, build_model, and
build_params gain a variant arg ("dcegm" | "brute").

The parity test asserts DC-EGM >= brute and allclose at the passive-
interpolation tolerance, excluding the lowest liquid-asset nodes where the
coarse consumption grid makes brute itself unreliable. It is skip-marked
(gpu-01 only) since it solves and OOMs the local box.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The splay/batch-size invariance assertions hard-coded float64 tolerances
(rtol/atol 1e-12 for the combo axes, 1e-9 for the stochastic-node reduction),
which the 32-bit CI job cannot meet: rescheduling the lax.map / reordering the
weighted-sum adds shifts results at single-precision scale. Key the tolerance
off conftest's X64_ENABLED, matching the established pattern in
test_economic_validation / test_grids.

Also stop test_egm_interp forcing dtype=jnp.float64 for its interpolation grid;
under --precision=32 that raises a float64-truncated-to-float32 UserWarning that
CI treats as an error. Use the canonical default float dtype instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The solve module's job is backward induction; name it for that. Renames
src/_lcm/solution/solve_brute.py → backward_induction.py and
tests/solution/test_solve_brute.py → test_backward_induction.py, updates
every import (model.py, simulation/compile.py, test_beartype_claw.py),
the two test-function names, and the doc/docstring references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hmgaudecker and others added 14 commits June 28, 2026 21:25
…oracle

Add a dense host VFI oracle (`tests/solution/_ds2024_housing_vfi_oracle.py`) that
mirrors the NEGM nest and searches the free-keep level `h(1-delta)` as an explicit
candidate alongside the outer house grid. At delta>0 that level is off the house
grid, so the on-grid brute twin cannot represent it — the oracle is the only valid
reference for the depreciating keeper.

Two interior tests:
- the oracle reproduces the brute solve at delta=0 (pins its economics);
- the NEGM solve at delta=0.10 reproduces the oracle (mean interior |dV| ~0.08,
  the same grid-resolution agreement as the delta=0 brute pair) — the keeper's
  depreciated hold is faithful.

Exclude the oracle helper from the test-naming hook, matching the other
tests/solution/_*_oracle.py helpers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…durable

A reviewer flagged that `_build_coh_shift_function` lifts adjusters using the
identity keeper reference (`next = durable`) rather than the keeper core's
no-adjustment level `keep(durable) = durable(1-delta)`, which the outer-envelope
design doc says should be the credited-cost reference. Switching to `keep(durable)`
in isolation is the design-exact shift, but a differential VFI-oracle test shows it
*regresses* DS-2024 delta=0.10: the corrected lift places each adjuster at a coh the
fixed keeper-grid envelope reads by extrapolation, and the one-island-per-adjuster
splice over-counts, moving the converging ~0.08 interior agreement to a plateaued
~0.18. The identity reference and the island splice are tuned together; the shipped
solver converges (mean 0.0825 -> 0.0759 as n_grid 10 -> 18). Correcting both belongs
with the segment-topology outer-envelope redesign, so document the coupling here
rather than ship a regression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Australia-2015-16 capital-income tax brackets are not printed in the DS-2026
paper text (only Figure 9); they are transcribed from the co-authors' replication
repo. Record the exact provenance so upstream drift is detectable: GitHub
akshayshanker/FUES, examples/housing_renting/config_HR/STD_RES_SETTINGS_4_TAXES/
master.yml, pinned at commit 00961e0b (blob 41661779), and a sha256[:16] checksum
(4e8d1bf0748f6933) of the resolved (LOWER, OFFSET, RATE) arrays. All 10 brackets
verified to match the source tax_table.brackets exactly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GPU peak-memory tracker spawns `python -m benchmarks.asv._gpu_mem` with
cwd=_PROJECT_ROOT, but _PROJECT_ROOT resolved to the benchmarks/ directory, not
the repo root, so the subprocess could not import `benchmarks` and every
GpuPeakMem benchmark failed with `ModuleNotFoundError: No module named
'benchmarks'` (the whole benchmark-pr workflow red). The file moved one level
deeper in 7dc5887 (benchmarks/ -> benchmarks/asv/) but the `.parent.parent` chain
was not updated. Point it at the true repo root (three parents up). Verified
locally: `python -m benchmarks.asv._gpu_mem` resolves from the corrected cwd and
fails only on missing argv, no longer on import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
W1 made `inverse_marginal_utility` optional: when a regime omits it, EGM derives
the inverse numerically (`numeric_inverse.py`; validation.py:298 documents this,
and tests/solution/test_iegm_wiring.py covers a model solving without it). So
removing that function is no longer a contract violation, and the negative case
in test_dcegm_contract_violation_raises stopped raising — failing `main`
deterministically on every platform. Drop the now-contradictory case; the iEGM
path's positive coverage already lives in test_iegm_wiring.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DC-EGM solve/oracle battery added on this branch makes the serial suite
~86 min on CI runners. Bake `-n 4` into the tests / tests-32bit / tests-with-cov
tasks so every CI job (and local `pixi run tests`) distributes across 4 workers;
`--dist loadfile` (already in addopts) keeps each file on one worker. Set
`XLA_PYTHON_CLIENT_PREALLOCATE=false` on the tasks — a no-op on CPU, but required
on GPU so the 4 worker processes don't each preallocate ~75% of the device and
OOM; they grow to their actual small per-test need, and loadfile keeps the one
memory-heavy file (Mahler-Yum) isolated to a single worker. Verified locally:
4/4 workers spin up and pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The state-action-param bundle threaded through `Q_and_F`, `max_Q_over_a`,
and their `_transform_and_average` / `_invert_joint` helpers carries every
flat param, including lookup-table params represented as `MappingLeaf` /
`SequenceLeaf` (tax schedules, pension accrual tables). The bundle was
annotated `FloatND | IntND | BoolND`, which excludes those leaf types, so
the package beartype claw rejected a `MappingLeaf` param whenever its O(1)
sampling happened to draw that key — an intermittent failure independent of
the model's correctness.

Widen the bundle annotation to the canonical `_ParamsLeaf` union. The two
concrete taste-shock extractions (`taste_shocks__scale`, `taste_shock_key`)
are narrowed back at their call sites with `cast`, since `logsum_and_softmax`
and `draw_taste_shock_noise` require the scalar/array types.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iles)

The single self-hosted GPU runner is starved of host CPU/memory when four
xdist workers each AOT-compile the DC-EGM oracle suite in parallel, and the
runner loses communication with the server (~60 min, then death) instead of
finishing the suite in roughly a quarter of that serially. Append `-n0` to the
two GPU steps so pytest-xdist's last `-n` wins and the suite runs in-process.
The CPU matrix keeps `-n 4`, where the four workers are a clear speedup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The macOS runner has 3 vCPUs and 7 GB of RAM; four xdist workers each
AOT-compiling the DC-EGM solve/simulate/oracle tests peak past its memory and
swap-thrash the suite to ~4 hours. A `pytest_collection_modifyitems` hook marks
that battery `slow` (all of `tests/solution/`, plus the DC-EGM-family modules
elsewhere), and the macOS matrix leg runs `-m "not slow"`. Linux and the GPU
runner still run the full suite, so the platform-independent kernel stays
covered; macOS keeps the lighter engine/regime/grid/simulation tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The committed `pixi.lock` is written in the format pixi 0.71.x emits; pixi
0.70.1 cannot parse it and aborts every job at install with
`expected a string, found table`. Pin the `setup-pixi` steps to the version
that produced the lock so the frozen install succeeds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`test_simulate_using_raw_inputs` calls `simulate` directly, bypassing the
public `Model.simulate` canonicalization, so its `flat_params` must already
carry canonical dtypes. `jnp.asarray(0)` is `int64` under x64, but the
params-leaf contract is `int32`; `_invert_joint`'s beartype check rejected it
whenever its O(1) sampling drew that key. Pin the leaf to `int32` — the dtype
`cast_params_to_canonical_dtypes` produces on the public path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base automatically changed from feat/type-local-continuation-v to main July 6, 2026 05:07
hmgaudecker and others added 12 commits July 6, 2026 07:43
Reconcile the two independent non-EU implementations that collided:

- Adopt main's first-class `CertaintyEquivalent` seam (`resolve_certainty_
  equivalent`, `ce.transform`/`ce.inverse`) throughout `Q_and_F.py`, dropping
  feat/dcegm's superseded `value_transform`/`inverse_value_transform` pair
  (`_build_continuation_operator`, `_transform_and_average`, `_invert_joint`,
  `_apply_continuation_operator`) and its test. DC-EGM keeps its own structure
  (`_get_deterministic_transitions` durable-carry, co-map, extended
  `_get_U_and_F`); the auto-merge's double-transform is removed.
- Thread `certainty_equivalent` through `processing.py` solve/simulate builders
  alongside the existing DC-EGM params (`solver`, `has_taste_shocks`).
- Repoint the DC-EGM custom-H admissibility check to main's default aggregator:
  `_default_H` (lcm.regime) → `H_linear` (lcm.temporal_aggregation), the object
  `finalize.py` now injects, so the identity check stays correct.
- Keep feat/dcegm's `period_kernels`/`build_lower_args` kernel API in the
  distributed HLO test (main's `max_Q_over_a` no longer exists on this branch).

ty + ruff clean; build-time DC-EGM validation, certainty-equivalent, Q_and_F,
custom-aggregator, and simulate-CE tests pass on CPU. DC-EGM×EZ solve behavior
must be confirmed on GPU CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 32-bit CI job failed 25 assertions whose tolerances (rtol down to
1e-12, atol down to 1e-12, decimal=10) are only meaningful at 64-bit.
Follow the existing `X64_ENABLED` convention: keep the 64-bit values and
use float32-eps-scaled bounds (typically 1e-5, with at least an order of
magnitude of margin over the observed 32-bit errors) otherwise.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Near the optimum the simulate argmax is nearly flat: the top-2
consumption-node values differ by less than the solve's cross-backend
variation, which enters at kink-adjacent nodes of the degenerate
last-alive-period EGM step (zero terminal continuation) and reaches a
few 1e-3 in V. Different backends and CPU microarchitectures therefore
legitimately settle on adjacent consumption-grid nodes; the windows CI
runner deterministically picked the neighbor of the pinned node.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two memory pathologies from running the DC-EGM battery in long-lived
pytest processes:

- ubuntu: JAX compile caches grow a worker's resident memory with every
  distinct compiled program, and four parallel workers running the full
  suite exhaust the 16 GB runner (the agent receives a shutdown signal
  mid-suite, seen twice at ~75%). Run the coverage job in three
  sequential invocations (fast set -n 4, then the slow battery in two
  fresh -n 2 processes) so each phase's memory is released; coverage is
  appended across them.

- GPU: the Mahler & Yum solve kernels only fit the T4 when XLA fully
  fuses the max-over-actions reduction. Compiled late in a process that
  has run the whole battery, the executable came out unfused, with a
  38.4 GB scratch arena (half the state-action product in float64) that
  can never fit a 16 GB device. The module now runs first, in its own
  process, pinning the fused compile; the identical program lowers to a
  0.5 GB arena in a fresh process.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The autoupdate commit dropped the lock file along with the rev bumps;
every frozen CI install needs it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…py use

- Widen the one-hot regime-transition wrapper's parameter annotations to
  the TransitionFunction protocol's union (parameters are contravariant,
  so the narrower `FloatND | IntND | int` was not assignable).
- Replace the deprecated `scipy.interpolate.interp1d(kind="cubic")` with
  `make_interp_spline(k=3)` — the same not-a-knot cubic spline, verified
  bit-identical on the knot layout the example uses.
- Pass `gauss_hermite` / `n_std` explicitly instead of through a
  heterogeneously-typed kwargs dict in the IID-process moment tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ty now runs as the official `astral-sh/ty-pre-commit` hook, so one
`prek run --all-files` covers all checks:

- The hook passes `--no-project` so uv neither creates a `.venv` nor
  resolves deps; ty resolves third-party imports from the pixi env named
  in `[tool.ty] environment.python` (the `type-checking` env, which keeps
  the stub packages but no longer carries the ty binary or a `ty` task).
- pre-commit.ci skips the hook (no pixi envs, no network at hook
  runtime); the run-ty CI job runs it against the `type-checking` env.
- Docs and agent instructions now name `prek run ty --all-files`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings the ty-via-prek type-checking section, the module overhaul
(distilled modules, numpy.md folded away), and the pandas provenance
rule. All @-references from this repo's AGENTS.md still resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
actions/checkout v6 -> v7, prefix-dev/setup-pixi v0.9.6 -> v0.10.0,
pixi v0.71.2 -> v0.72.0, codecov/codecov-action v6.0.1 -> v7.0.0.
actions/setup-python stays on the v6 major and
pypa/gh-action-pypi-publish on its rolling release/v1 tag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Addresses the findings from the PR #390 code review.

Correctness:
- step_core: NaN-poison the carry marginal-utility row on envelope
  overflow, matching the value rows, so an overflowed period cannot feed
  the parent a finite-but-wrong Hermite slope past the NaN diagnostics.
- numeric_inverse: require finite bracket marginals in the log_well_defined
  gate, so a +inf marginal utility (steep CRRA at a near-zero c_lower)
  fails loud instead of running the log path on a non-finite endpoint.
- FUES upper envelope (#387): scan exhaustively by default
  (n_points_to_scan=None -> every candidate). A bounded window silently
  accepts dominated points when more than the window's worth of off-segment
  candidates interleave between two points of one segment. An explicit finite
  width remains available as a speed-vs-correctness opt-in. The F4 strict
  xfail becomes a passing test; a companion test pins the bounded-mode
  tradeoff.

Cleanup and documentation:
- regime_template: delete the dead value_transform/inverse_value_transform
  special-case orphaned by the certainty-equivalent merge.
- validation: correct the module and helper docstrings that overclaimed the
  grid rules (batch_size is honored; only distributed is rejected) and that
  wrongly listed inverse_marginal_utility as required.
- asset_row: docstring "weakly ascending" -> "strictly increasing" to match
  the enforced resources-monotonicity check.
- continuation: document why a device-sharded carry into a DCEGM parent is
  unsupported, that three existing rules already make it unconstructible, and
  concrete ideas for lifting the restriction. A regression test pins the fence.

Tests: ty, ruff, and the EGM/DCEGM/simulation suites pass (525 passed,
3 skipped); FUES parity suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123hk3uzBgBeZown4eBBQ2C
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FUES bounded scan can accept a run of dominated points (n_points_to_scan default under-scans)

1 participant